Web applications that serve and manage millions of Internet users, such as Facebook™, Instagram™, Twitter™, banking websites, as well as online retail shops, such as Amazon.com™ or eBay™ are faced with the challenge of ingesting high volumes of data as fast as possible so that the end users can be provided with a real-time experience.
The “Internet of Things” (IoT) is another major contributor to big data, supplying huge volumes of data. IoT has become a pervasive presence in the environment, with a variety of things/objects that communicate via wireless and wired connections to interact with each other and cooperate with other things/objects to create new applications/services. These applications/services exist in smart cities (regions), smart cars and mobility, smart homes and assisted living, smart industries, public safety, energy and environmental protection, agriculture and tourism. A massive quantity of data gets persisted from the millions of IoT devices and web applications.
In current event recordation systems, a high volume of events is published by multiple event producers and then delivered to multiple consumers. Events are published to an event recordation system with a replay identifier (replay ID). The replay identifier is representative of the order of event publishing. Each event is published through execution of a transaction that is processed prior to the event being committed in the event recordation system. The order by which events are committed to the event recordation system can be different from the order identified by the replay ID.
The distribution/delivery of the events to the event consumers is performed in a sequential order based on the replay IDs such that a first event with a replay ID smaller than the replay ID of a second event needs to be delivered to a consumer prior to the second event regardless of when the first event was stored in the event recordation system when compared with the second event. However, given that events can be stored in the event recordation system in an order that is different from the publication order of the events, the second event can be stored and therefore made available to consumers prior to the first event. In these scenarios if the consumers consumer the second event that results in the first event never being delivered to the event consumers.